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ABOUTTHE BEAMS PROJECT 

During 2003-07, the Building Engagement and Attainment for Minority 
Students (BEAMS) project fostered data-based campus change initia- 
tives at more than 100 four-year Historically Black, Hispanic-Serving, 
and Tribal colleges and universities to increase student engagement 
and learning. Each campus made a commitment to analyze the scope 
and character of its students’ engagement by participating in the 
National Survey of Student Engagement (NSSE) and to implement 
well-designed action plans intended to improve student engagement, 
learning, persistence, and success. The Summer Academy — an annual 
gathering of representatives from various colleges and universities for 
collaborative work aimed at increasing access and success in higher 
education — provided dedicated time for BEAMS teams to identify 
solutions that could influence institutional and national higher educa- 
tion policy and practices. BEAMS is a partnership between NSSE 
and the Alliance for Equity in Higher Education, which is managed by 
the Institute for Higher Education Policy, and is supported by Lumina 
Foundation for Education. 

This practice brief is one of a series highlighting key practices under- 
taken by some of the many successful BEAMS schools during the 
project’s five years of data collection and action plan implementation. 
These practice briefs accompany a monograph that details the 
process BEAMS institutions used to craft data-driven action plans 
and to implement those plans to improve student success. The 
purpose of the practice brief series is to outline effective practices 
that can be replicated in postsecondary institutions interested in 
pursuing data-based change and increasing student engagement, 
learning, and success. This particular brief provides suggestions 
for successful survey administration and data collection based on 
findings from the BEAMS project to show how assessment can be 
used to increase student engagement and success. 



INTRODUCTION 

The need to use evidence to improve and strengthen the quality of post- 
secondary education has never been greater. In an era when national 
assessment measures and voluntary systems of accountability and 
transparency are being proposed in higher education, it is essential to 
maximize the effectiveness of all assessment efforts by using high-quality 
data. It is particularly important that Minority-Serving Institutions (MSIs) 
are able to demonstrate their effectiveness with evidence and ensure 
that their distinct mission is reflected in assessment activities. 

The suggestions for assessment offered here are based on a review 
of the data collection approaches used by BEAMS project schools. 
NSSE measures the degree to which first-year students and seniors 
participate in effective education practices, and all of these institutions 
used this instrument to assess and improve student engagement and 
success. Lessons for a successful survey administration are drawn to 
help other institutions address the challenges of data collection. 
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ASSESSMENT AND THE BEAMS PROJECT 
Assessment in higher education is a process that involves the planned 
examination of information and practice related to student learning. 
When done well, assessment helps colleges establish a culture of evi- 
dence wherein data about student learning informs institutional actions 
and decisions as well as long- and short-term planning. The quality of 
any assessment initiative depends on many factors, but once an institu- 
tion commits to its assessment plan, one of the most important first 
steps is data collection. 

Institutions in the BEAMS project committed to conducting two NSSE 
data collections in their assessment efforts. The first provided baseline 
information about the quality of the undergraduate experience. BEAMS 
team members spent time reviewing and interpreting NSSE results 
in combination with other information about student success on their 
campus, including retention and completion rates. This information 
informed the development of a campus action plan. The second NSSE 
administration offered a formative assessment of the campus initiative 
and provided early measures of the impact of the project on student 
engagement and success. 

From the outset, BEAMS schools had different levels of experience using 
assessment tools. Some had long histories of administering locally 
developed placement tests and surveys to assess satisfaction, and 
several had experience using NSSE. Few, however, had established 
a culture of using evidence to inform decision-making, and even fewer 
had experience using standardized instruments. Most schools had 
limited capacity to collect data on student experiences, and few 
had effective mechanisms to link data to campus change efforts. 

At the core of some of the most underdeveloped assessment cultures 
was legitimate skepticism about the validity of standardized instruments 
for use with minority students and the concern that assessment data 
would simply be used to identify and punish poor performers. NSSE’s 
history as a useful tool for institutional improvement and survey testing 
with diverse populations, as well as one that rejects the use of data for 
ranking, lessened these concerns. 

Tests with students at MSIs and Predominantly White Institutions (PWls) 
showed that NSSE works equally well for students from different racial 
and ethnic backgrounds attending different types of institutions. This work 
affirms the construct validity of NSSE — that it consistently measures what 
it claims to measure. In addition, almost all NSSE items are generally 
interpreted in the same way by students from different racial and ethnic 



backgrounds. Thus, NSSE can be considered a valid, reliable indicator of 
the quality of the student experience for historically underserved students 
at MSIs and PWls. 

Data collection was complicated at BEAMS schools, particularly those 
that lacked the technical capacity to administer NSSE. A lack of 
university-issued e-mail addresses or reliable Internet systems made 
online survey administration undependable at some schools, while other 
schools did not have complete postal addresses to conduct a paper 
survey administration. Schools that conducted local administrations 
had difficulty appointing staff to coordinate and administer the survey. 
Although NSSE’s standard protocol is designed to ease data collection 
efforts, it proved challenging at some under-resourced schools. 

Despite these challenges, most BEAMS schools eagerly undertook 
their NSSE administration. They were enthusiastic about the potential 
to demonstrate their good work with evidence and looked forward to 
the opportunity to use data to create action plans. The most successful 
schools planned their administration strategy, involved faculty and stu- 
dents throughout the NSSE administration, implemented context-specific 
publicity and incentives for participation, and cultivated widespread 
interest in the survey process and results. The design of a successful 
NSSE administration required attention to the following: 

• Selecting a survey administration mode appropriate for the campus 

• Proactively addressing the technology of survey administration 

• Identifying staff to oversee the administration 

• Publicizing the survey and promoting the value of the results 

• Involving students and faculty in administration decisions and efforts 
to publicize the survey from the outset 

• Cultivating a systemic, data-informed approach to studying under- 
graduate education by using various measures of student learning 

Ultimately, a successful survey administration is characterized by a 
process that ensures the collection of information that is accurate, 
representative, meaningful, and actionable. Although high response 
rates are often used as a measure of a successful administration, there 
is no standard acceptable response rate. Rather, the most important 
consideration is that the data are representative of the population being 
assessed. Without a good representative sample, no result can be 
considered valid regardless of the response rate or sample size. Attention 
to the survey administration process is the best way to ensure represen- 
tative and meaningful results. 
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STEPS TO SUCCESSFUL SURVEY DATA COLLECTION 
This section outlines steps that BEAMS schools took before, during, 
and after their NSSE administrations to ensure successful data collection. 
These steps are important in any campus data collection effort. 

Select an appropriate survey mode. One of the first steps in 
successful data collection is the selection of the appropriate survey 
mode. BEAMS schools chose from among four modes for 
administering NSSE: 

• Web-only (students receive all correspondence by e-mail and 
complete the Web survey) 

• Web plus (students receive first contacts via e-mail, and a sample 
of non-respondents receive a paper survey via postal mailing) 

• Paper (students receive paper survey in postal mailings) 

• Local administration (institutional representatives distribute and 
collect surveys) 

The advantage of using the two Web-based and paper administration 
modes is that the administration protocol is standardized and handled 
by NSSE. In the local administration mode, the institution is responsible 
for all data collection activities. 

BEAMS schools that had a dependable technology infrastructure elected 
to use a Web-based administration. This mode requires a reliable e-mail 
system and good survey promotion. BEAMS institutions used their campus 
Web portals to publicize the survey before and during their survey admin- 
istration. Several institutions created a link on their university homepage 
to the survey and others created an opening page with a short prompt 
encouraging students to complete the NSSE. New Jersey City University 
(New Jersey), Winston Salem State University (North Carolina), and 
Fayetteville State University (North Carolina) promoted their survey 
administration on the homepage of their Web sites, and several schools 
put the NSSE announcement on screen savers in computers in the 
library and in labs. 

Paper administrations were used at BEAMS schools that had reliable 
campus postal mail addresses. Fewer institutions used this approach 
than others, but it worked well for Chicago State University (Illinois) and 
the smaller, more residential University of St. Thomas (Texas). These 
institutions used the standard NSSE reminders and postcards and 
additional posters in residence halls to promote the survey. 



BEAMS schools that lacked the infrastructure to conduct a standard 
Web-based or paper administration used the local administration option. 
Because this option requires the institution to coordinate and administer 
the survey, it only worked well for schools with staff who were strategic 
about opportunities for data collection. Johnson C. Smith University 
(JCSU) in North Carolina opted for local administration because they 
had direct ways to administer the survey. They scheduled their NSSE 
administrations well in advance, and included dates in the JCSU course 
catalog. They administered the survey to first-year students via their 
Freshman Academy and to seniors at convocation events. Most important, 
their data collection activities were enhanced by connecting their NSSE 
results to their Southern Association of Colleges and Schools reaffirmation 
and, specifically, to their Quality Enhancement Plan, which detailed a 
plan to strengthen learning in the first year. JCSU staff achieved a high 
participation rate because they identified convenient opportunities for 
data collection, promoted the survey among students and faculty, and 
emphasized the value of the survey in their accreditation work. 

Publicize the survey. A crucial activity in a successful survey adminis- 
tration is promoting the survey to its intended respondents and others. 
Survey publicity such as flyers and media articles as well as incentives 
for survey completion can communicate to the campus that the data are 
valuable. The promotional approaches used by BEAMS schools described 
here were supplementary to the standard invitations to participate in 
NSSE. Direct contact with students who are invited to participate in the 
survey must be conducted in accordance with human subjects protocol 
and should not cause undue influence on participation. 

The most commonly implemented forms of publicity used by BEAMS 
schools were in-class announcements by faculty emphasizing the 
importance of the survey and flyers or posters in public areas, including 
student centers, dining halls, and residence halls. Some campuses 
placed announcements in the campus newspaper and in e-news 
listings. Institutions did the most publicity right before actual survey 
administration, and many actively promoted the importance of partici- 
pation when the survey was in the field. Some schools asked student 
government, fraternities and sororities, and academic clubs to remind 
members about the importance of responding to an invitation to complete 
NSSE. In the best cases, student leaders were involved in the NSSE 
administration and were later involved in the interpretation and use 
of NSSE results. 
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Norfolk State University (Virginia) publicized its participation in NSSE 
by putting up posters and table-tents in the student center. They took 
advantage of NSSE’s Web interface to monitor student participation. 
This tool provided Norfolk with information about non-responders and 
participation patterns. For example, they learned which forms of public- 
ity had an impact when they saw upticks in response rates immediately 
after NSSE e-mail reminders were distributed and new table-tents were 
posted in the student center. When the interface revealed two days of 
non-response, they knew it was time for additional promotion. Norfolk 
staff members and influential student leaders then spent a day in 
the student center publicizing the survey to all students, giving away 
reminder notes attached to candy and other Norfolk State trinkets. 
Other institutions that used the Web interface to monitor participation 
saw increases in survey response when they placed posters on campus 
buses and ran spots about the survey on the campus radio station. 

Build a support base for assessment. Finally, the successful 
collection and use of survey data depend on building a foundation of 
support for assessment, creating awareness and investment among all 
members of the campus community, and communicating clearly about 
how the data will be useful in efforts to improve student success. Because 
the BEAMS project was set up to involve a range of participants, there 
was a basis for such support. However, the most successful institutions 
expanded this foundation to promote their NSSE administration. 

For example, the student affairs division at St. Thomas University 
(Florida) made significant progress toward developing a culture that 
values assessment, honors student input, and supports a partnership 
between faculty and student affairs. The St. Thomas BEAMS team saw 
their NSSE administration as an opportunity to unite faculty, students, 



and student affairs staff to plan ways to use the results to inform the 
restructuring efforts already under way in the Division of Student Affairs. 
They involved students, faculty, and staff in the promotion of their NSSE 
administration and were explicit in promotions about how the data 
would be used to improve the quality of the student experience. Results 
helped student affairs staff advocate for the L.I.F.E.L.O.N.G. Center for 
Leadership and Student Engagement, a home for courses, workshops, 
experiential learning exercises, online resources, and developmental 
opportunities that build on the existing strengths and talents of students. 

CONCLUSION 

Knowing how to improve institutional effectiveness is nearly impossible 
without good information about the quality of the student experience. 
Surveys provide an important source of this assessment information. 

Yet, student participation is critical to survey success. More respondents 
yield better quality data. Sampling error, total completions, and response 
rates are crucial measures of data quality and are important for increasing 
confidence in the results among a wider audience. 

A healthy culture for assessment develops when everyone on campus 
feels comfortable sharing their views with the knowledge that information 
will be taken seriously and will be used to improve the learning environment. 
The positive impact of student engagement results are multiplied when 
data are relevant to faculty, staff, and students. Good data collection is 
just the first step in equipping institutions with the information they need 
to improve the conditions for student success. Ultimately, the likelihood 
that changes in policy and practice will succeed will increase when 
campus partnerships form at the start of the data collection process 
and early decisions are made about the strategic use of data. For more 
information on NSSE, visit http://nsse.iub.edu. 
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